DATA ENGINEERING / BATCH + CDC PIPELINE

Replay anything.
Lose nothing. Duplicate nothing.

VeloCiSync M2MR is a dual-path pipeline where Spark replays historical GitHub-event backfills through MySQL staging while Debezium and Kafka stream canonical database changes into MongoDB and Redis — engineered so that re-running any step never corrupts the result.

View source code ↗

ROLE

Data Engineer

SCOPE

Individual project · Jan – Mar 2026

DATA

438,018 GitHub events · 24 hourly partitions

STACK

Python · Spark · MySQL · Debezium · Kafka · MongoDB · Redis · Docker

01 / CONTEXT & PROBLEM

Backfills and live sync fail in opposite ways.

Historical backfills get re-run — after a crash, a bug fix, a partial load — and naive re-runs create duplicates. Live CDC has the opposite failure: if the consumer commits its Kafka offset before a downstream write lands, a crash silently loses events. Most tutorial pipelines handle neither.

The goal: one system where MySQL is the single canonical store, historical JSONL replays are provably idempotent, and CDC projections into MongoDB and Redis are exactly-once in effect — with the audit trail to prove it.

02 / ARCHITECTURE

Two paths, one canonical truth

PATH A · HISTORICAL BACKFILL (solid)

GitHub events JSONL

438,018 events · 24 hourly partitions

→

Spark processing

contract validation · rejected-record quarantine

→

MySQL staging

SHA-256 fingerprints · idempotent merges · dedup

→

MySQL canonical

immutable event IDs · audit accounting

PATH B · NEAR-REAL-TIME CDC (dashed)

MySQL binlog

canonical changes

→

Debezium → Kafka

change topics

→

Checkpoint-safe consumer

offset commits ONLY after both targets succeed

→

MongoDB + Redis

document & low-latency projections

Entire stack containerized with Docker Compose · Makefile automation · CI validation · operational runbooks · 14 unit/integration tests

03 / KEY DECISIONS

What I chose, and why

Idempotency by construction, not by cleanup

Immutable event IDs + SHA-256 payload fingerprints + staged merges mean any partition can be replayed at any time — duplicates are detected and conflicts protected at write time, never repaired afterwards.

Offsets advance only after BOTH projections land

The CDC consumer treats MongoDB and Redis as one atomic outcome: a crash between writes replays the record instead of losing it. At-least-once delivery + idempotent writes = exactly-once effect.

MySQL as the single canonical store

MongoDB and Redis are disposable projections, rebuildable from MySQL at any time. That one rule collapses the hardest distributed-systems questions into "replay the log".

Audit accounting as a first-class output

Every batch reports accepted / rejected / duplicate counts that must reconcile to input totals — the same discipline as financial bookkeeping, applied to events.

04 / RESULTS

Measured outcomes

438,018

events profiled & processed across 24 partitions

17,786

event backfill validated end to end

46.33/s

CDC throughput on a 1,000-record local batch test

14

unit/integration tests + CI + runbooks + Makefile

An analytics notebook on top covers activity volume, actor/repository concentration, hourly patterns, and KPI quality gates with dashboard-ready outputs.

05 / WHAT I'D DO DIFFERENTLY

Measure throughput under contention, not just correctness.

The 46.33 rec/s figure comes from a single-consumer local batch test. The next iteration gets consumer-group parallelism with per-partition ordering guarantees, plus load tests that measure how the checkpoint-safe pattern behaves when one target (Redis vs MongoDB) is degraded.

I'd also promote the audit accounting into queryable tables with alerting, so reconciliation failures page someone instead of living in logs.